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ABSTRACT 



Since early 1998, AEL, Inc., has been assisting state 



departments of education in Kentucky, Tennessee, Virginia, and West Virginia 
in the implementation of the federally funded Comprehensive School Reform 
Demonstration (CSRD) program. As part of this assistance, AEL established two 
types of electronic networks to link school staff at CSRD sites, external 
facilitators, and site researchers assisting with school formative 
evaluation. CSRDnet is a Web-based electronic communication forum providing 
moderated discussion and conferencing capabilities for external facilitators 
and CSRD sites. Participants are assigned to discussion groups based on their 
state and the CSR model they are implementing. The second electronic medium 
consists of listservs in Kentucky, Virginia, and West Virginia. As with 
CSRDnet, a moderator facilitates discussion and information sharing, and 
participants include external facilitators and staff at CSRD sites. 
Evaluations of these networks included analyses of CSRDnet activity and 
listserv activity during January-March 2000 and surveys of client 
satisfaction completed during 1999. Of 96 individuals responding to questions 
about the forums, 55 were located in Kentucky. Results suggest that CSRDnet 
and the listservs are underutilized. Survey respondents were satisfied with 
the format, timeliness, and efficiency of CSRDnet and the knowledge and 
preparedness of AEL staff, but were less satisfied with the usefulness of 
CSRDnet to their work and its relevance to their particular situations. 
Appendixes include the client satisfaction survey. (SV) 
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INTRODUCTION 



Federal CSRD Legislation 

School improvement is increasingly viewed as an ongoing and comprehensive process. 
Recent legislation has encouraged the adoption of such a view: In 1994, Congress revised the 
regulations to allow schools where 50% or more students qualify for free or reduced lunch to use 
their Title I funds for whole school improvement rather than for programs targeted solely at specific 
groups of students (American Institutes for Research, 1 999). Later, as part of the FY 98 Labor-HHS- 
Education Appropriations Act (PL 105-78), and again in 1999, Congress designated $150 million 
to promote comprehensive school reform. Much of these Comprehensive School Reform 
Demonstration (CSRD) funds flowed through states via a competitive grant process to schools and 
districts interested in implementing schoolwide, comprehensive reform. Of these monies, $120 
million came from Elementary and Secondary Education Act (ESEA) Title I funds allocated by 
formula to states to allow current Title I schools to adopt comprehensive school reform models. The 
Fund for the Improvement of Education (FIE) allocated $25 million to flow to states by school-age 
population to allow non-Title I schools to implement reform programs. In addition, $4 million was 
allocated to Regional Educational Laboratories to assist states, districts, and schools in the 
implementation of the CSRD program. 

According to the New American Schools (NAS) organization (2000), which has developed 
several of the schoolwide reform models adopted by CSRD-funded schools, “the success or failure 
of the CSRD Program has enormous implications for public education in the country’s poorest 
neighborhoods.” Title I of the ESEA of 1965, which provides federal funding for disadvantaged 
students, is due for congressional reauthorization in the year 2000. “To encourage research-based 
and replicable best practices,” Congress may reauthorize Title I based upon lessons learned from 
CSRD (NAS, 2000). 



AEL CSRD Program 

AEL, Inc. began assisting the four states in its region, Kentucky, Tennessee, Virginia, and 
West Virginia, in early 1998 as part of the effort to support state departments of education in their 
implementation of CSRD. Assistance to these state departments of education has included 
consultation and technical assistance concerning the development of procedures and criteria to select 
competitively the schools that would receive assistance under this program. Also, AEL has 
collaborated with state departments of education in developing procedures for guiding interested 
districts and schools though the process of assessing their needs, selecting among research-based 
reform models, or choosing to develop their own research-based comprehensive program. Also in 
collaboration with state departments, AEL has developed and provided workshops and technical 
assistance to schools and districts preparing their applications for grants under this program. Such 
workshops have included information about candidate reform programs, assistance with compiling 
aggregate components to create comprehensive programs for those not opting to purchase a model, 
and guidance regarding how to evaluate the effectiveness of programs implemented. 
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Another significant component of AEL’s CSRD program is an Academy for External 
Facilitators. External facilitators are individuals assigned by state departments of education to assist 
schools implementing CSRD-funded reform initiatives. AEL’s Academy offers training and support 
as external facilitators execute five challenging roles in school reform: using the knowledge base on 
effective practices leading to improved student performance; modeling collaboration and joint 
problem-solving; collecting, organizing, and analyzing school data to make decisions; brokering and 
recommending resources; and acting as agents of change in contexts in which facilitators possess 
no authority to direct change. 

Another facet of AEL’s CSRD program has been to assist states and schools evaluate the 
effectiveness of programs implemented with CSRD funds. Participating CSRD schools are required 
to evaluate their progress toward the nine CSRD implementation criteria (evaluation itself being one 
of the nine criteria). 

AEL CSRD services have additionally included a research symposium on the CSRD program 
and schoolwide reform and several independent and collaborative research efforts. 

Electronic Networks 

Yet another AEL CSRD goal is to develop electronic networks of schools, districts, and 
external facilitators involved in CSRD. The purpose of such networks is to provide participants with 
opportunities to share experiences and information, connect with resources, and access relevant 
information. Moreover, the electronic networks provide AEL a venue for offering ongoing support 
to CSRD school sites and external facilitators. These networks are also intended to provide AEL 
with information regarding contributions to successful implementation of CSRD programs for the 
purpose of informing the wider education community. 

AEL has established two types of electronic networks in which staff at CSRD sites, external 
facilitators, and site researchers assisting with school formative evaluation may participate. CSRDnet 
is a web-based electronic communication forum providing moderated discussion and conferencing 
capabilities for external facilitators and CSRD sites. Participants are assigned to several discussion 
groups based on the CSR model they are implementing and the state in which they work. A form is 
provided for participants to enter information about themselves and their undertakings; a similar 
form is provided for participants to complete profiles of their schools. In addition, a moderator posts 
articles and links relevant to CSR. 

The second electronic medium consists of listservs in three of AEL’s four states (Kentucky, 
Virginia, and West Virginia). As with CSRDnet, a moderator facilitates discussion and information 
sharing, and participants include external facilitators and staff members at CSRD sites. 

This evaluation report is intended to provide AEL CSRD staff with a description of the extent 
to which and the ways in which CSRDnet and the three listservs has been used between January 5 
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and March 10, 2000. It is also intended to provide information to staff as they plan modifications to 
the electronic forums. 

METHODS 

Several methods are used to evaluate AEL’s CSRD electronic networks. CSRDnet activity 
is logged weekly into a SPSS database. The number of log-ins, entries, and files accessed are 
counted each week; staff and moderator activity is then subtracted from weekly counts to render a 
more accurate account of external facilitator and CSRD site participation on CSRDnet. Data 
collection via this method began January 5, 2000. 

A second source of information is data from the 1999 survey of client satisfaction with 
AEL’s CSRD services (see Appendix A), administered to state department, local school district and 
local school personnel who have availed themselves of AEL CSRD services. This survey is intended 
to assess clients’ degree of satisfaction with a variety of characteristics of the services they have 
received. Respondents are requested to rate their level of satisfaction with 12 characteristics of the 
services they obtained using a 5-point Likert-type scale. In addition, respondents are asked three 
open-ended questions which seek to identify suggestions for improving current services, clients’ 
needs for the coming year, and ways AEL CSRD services have been useful. 

Two versions of the survey are administered: one to state department partners, and the other 
to local school district and school faculty. Items are tailored to address the AEL CSRD services each 
role group might have accessed. Data reported in this account are drawn from the version 
administered to local school district and school personnel. 

In addition, several demographic items are included on the local district and school version 
of the satisfaction survey. Role, state of employment, school level, and number of years in education 
are among the demographic characteristics solicited. 

The satisfaction survey was intended to be a population survey of local district and school 
personnel who had received technical assistance or consultation with AEL staff about CSRD during 
1 999. Contact information for district and school personnel participating in the Formative Evaluation 
Process for School Improvement (FEPSI) are maintained in a database; 127 schools in AEL’s region 
receive CSRD funding and have been listed in the database. Of the 1 27, 96 also participate in FEPSI. 
This database also includes contact information for individuals from five schools not funded by 
CSRD which are nonetheless participating in FEPSI, and 30 CSRD-funded schools not electing to 
participate in FEPSI. The database contains data on a total of 1 3 1 schools. 

However, participant lists with complete contact information were not collected at every AEL 
CSRD workshop or meeting conducted during 1999. To assist in creating a more accurate sample, 
project staff requested from each state department of education in AEL’s region lists of participants 
who had attended workshops or technical assistance sessions collaboratively developed and/or 
presented by AEL and the SEAs during 1999. Similarly to the FEPSI database, these participants 
were local school and district employees. Missing contact information from such participant lists was 
then sought by evaluation staff via the Quality Education Data (QED) database purchased by AEL 
providing information about local district educators in the region. 
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Information was not available from the QED database for every individual, however. This 
is likely because QED data were collected during the prior school year, and some educators may 
have been employed elsewhere. 

A total of 422 surveys was mailed February 23, 2000. Three weeks later, a reminder card was 
sent to those participants who had not yet returned their completed surveys to AEL evaluation staff. 
A second reminder card was mailed four weeks following the first. By May 2000, a total of 222 
surveys had been completed and returned, for a return rate of 53%. 

Section 7d of the instrument asks respondents to rate their satisfaction with 12 characteristics 
of CSRDnet on a 5-point scale, ranging from 1 (very dissatisfied) to 5 (very satisfied). Demographic 
data allows disaggregation of findings. In addition, respondents’ replies to three open-ended 
questions about AEL’s CSRD services were analyzed with regard to the electronic forums. 

Third, listserv activity is monitored and stored in an e-mail application. Postings from 
participants are then coded and tabulated. Data reported in this account were collected between 
January 1 and March 10, 2000. 



FINDINGS 
Log of CSRD Activity 

Over the course of the first 1 8 weeks of the year 2000, CSRDnet activity was logged into an 
SPSS database. Log-ins, entries, and files accessed are counted each week; staff and moderator 
activity is then subtracted from weekly counts to present a more precise account of external 
facilitator and CSRD site participation on CSRDnet. 

The mean number of log-ins per week minus staff and moderator log-ins was 7.17, with a 
standard deviation (SD) of 6.44. The number of weekly log-ins by clients varied from none to 20. 
The mean number of entries made or edited by clients was 2.17 (SD 3.60), and the mean number of 
files accessed weekly by clients was 3.50 (SD 4.66). Descriptive statistics are presented in Table 1. 

Table 1 

Descriptive Statistics for CSRDnet Activity 



Item 


N 


Minimum 


Maximum 


Mean 


SD 


Total log-ins 


18 


ii 


45 


21.78 


9.60 


Total client log-ins 


18 


0 


20 


7.17 


6.44 


Total entries 


18 


0 


39 


9.11 


11.71 


Total client entries 


18 


0 


12 


2.17 


3.60 


Total files accessed 


18 


0 


16 


4.44 


5.09 


Total files accessed by clients 


18 


0 


16 


3.50 


4.66 
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Standard deviations were relatively high, suggesting that the number of log-ins, entries, and 
files accessed varied substantially across weeks. Analysis of the data file indicated that the heaviest 
activity occurred during the week of January 5-12, 2000, with a total of 20 client log-ins, 12 client 
entries, and 6 files accessed by clients. The week of February 10-16, 2000, was the second most 
active week on CSRDnet, with 20 client log-ins, 8 client entries, and 2 files accessed by participants. 
The least client activity occurred during the week of April 27-March 3, 2000. No clients logged into 
CSRDnet during this week. 



1999 Annual Satisfaction Survey 

Two-hundred twenty-two individuals engaged in CSRD reform efforts completed and 
returned the local district and school faculty version of the 1999 annual satisfaction survey. Of these, 
96 (43%) responded to at least one of the items concerning their satisfaction with CSRDnet. More 
than half (57.3%) of such respondents work in Kentucky, and more than half (51.1 %) are employed 
in elementary schools. Table 2 shows in which states respondents are employed. Table 3 provides 
information on the school levels at which respondents work. 



Table 2 

Respondent Location 



State 


N 


Percent 


Kentucky 


55 


57.3% 


Tennessee 


9 


9.4% 


Virginia 


14 


14.6% 


West Virginia 


18 


18.8% 



Table 3 

Respondent School Level 



School Level 


N 


Percent 


Elementary 


48 


r i i A/ 

Di.iVo 


Middle 


11 


11.7% 


High School 


4 


4.3% 


Multiple levels 


31 


33.0% 



Respondents played a variety of roles in schools. Nearly a third (30.2%) identified 
themselves as school administrators, and more than a fourth (27.1%) as external facilitators. Table 
4 summarizes the roles held by respondents. 




IQ 
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Respondent Role 
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Role 


N 


Percent 


School administrator 


29 


30.2% 


External facilitator 


26 


27.1% 


School staff 


20 


20.8% 


Local school district central office staff 


16 


16.7% 


Other 


4 


4.2% 


CSR model developer/staff 


1 


1.0% 



Nearly two-thirds (65.6%) of respondents reported having more than 20 years of experience 
in education. Respondents varied widely in the number of years they had spent working in their 
current location. Tables 5 and 6 display descriptive statistics concerning respondents’ years of 
experience in education and working in their current sites. 



Table 5 

Years Employed in Education 



Years in education 


N 


Percent 


1-3 years 


1 


1.0% 


4-10 years 


4 


4.2% 


11-20 years 


28 


29.2% 


More than 20 years 


63 


65.6% 



Table 6 

Years Working in Current Location 



Years working in current location 


N 


Percent 


Less than 1 year 


9 


9.5% 


1-3 years 


30 


31.6% 


4-10 years 


26 


27.4% 


11-20 years 


16 


16.8% 


More than 20 years 


14 


14.7% 
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Overall, respondents expressed less satisfaction with characteristics of CSRDnet than with 
characteristics of any other AEL CSRD service. Of the 222 respondents, only 96 (43%) replied to 
items about CSRDnet, using a 5-point Likert-type scale with anchor points of 1 (very dissatisfied) 
and 5 (very satisfied). Respondents reported the highest amount of satisfaction with the preparedness 
(4.00, SD 1 .00) and knowledge (3.99, SD 1 .03) of the CSRDnet service providers, and the format of 
the medium (3.97, SD 1.02). Also highly rated was respondents’ satisfaction with the timeliness 
(3.95, SD 1.06) and the efficiency (3.93, SD 1.04) of the service delivery. Table 7 presents mean 
ratings of satisfaction with 12 characteristics of CSRDnet. 

The usefulness of CSRDnet (3.57, SD 1.10), the extent to which it met respondents’ needs 
(3.61, SD 1.00), and its relevance to respondents local or state situations (3.68, SD 1 .06) received the 
lowest mean ratings. The relatively large standard deviations suggest, however, that there is some 
divergence among respondents about their levels of satisfaction. 



Table 7 

Mean Ratings of Satisfaction with CSRDnet Characteristics 



Item 


N 


Mean 


SD 


Preparedness of service provider 


93 


4.00 


1.00 


Knowledge of service provider 


92 


3.99 


1.03 


Format of service delivery 


95 


3.97 


1.02 


Timeliness of service delivery 


94 


3.95 


1.06 


Efficiency of service delivery 


95 


3.93 


1.04 


Overall satisfaction with this AEL CSRD service 


96 


3.77 


1.03 


Applicability of information received 


95 


3.77 


0.99 


Adequacy of materials provided 


94 


3.76 


1.04 


Completeness of information provided 


94 


3.73 


1.01 


Relevance of service to your state/local situation 


94 


3.68 


1.06 


Extent to which service met your needs 


93 


3.61 


1.00 


Usefulness of service to your work 


95 


3.57 


1.10 



Analyses of variance (ANOVA) were conducted to determine if differences in mean ratings 
of satisfaction with CSRDnet among respondents were statistically significant. This test of statistical 
significance allows determinations of whether differences are likely due to variations occurring by 
chance or due to actual differences in the groups under study. ANOVAs were used here to detect 
statistically significant differences among respondents based on their role, school level, state, years 
of experience in education, and years of experience in their current location. Five statistically 
significant differences were found at the .05 level of statistical significance (see Table 8). 
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Item 


df 


F ratio 


F 

probability 


Significant 
differences by 
group 


By role 
group 


Relevance of the service to your state/local 
situation 


5 

88 


2.42 


.04 


N/A 


Adequacy of materials received 


5 

88 


3.46 


.01 


N/A 


By 

school 

level 


Efficiency of service delivery 


3 

89 


2.70 


.05 


multiple levels > 
high school 


Completeness of information provided 


3 

88 


2.90 


.04 


multiple levels > 
high school 


Knowledge of service provider 


3 

86 


3.25 


.03 


multiple levels > 
high school 



Two statistically significant differences were detected among role groups in terms of their 
mean ratings of satisfaction with the relevance of CSRDnet to their local or state situations and the 
adequacy of materials provided. Tukey post hoc tests were conducted to determine among which of 
the role groups the statistically significant differences were located. However, the post hoc tests could 
not be performed because at least one role group contained too few cases. 

Cross tabulation of means for these items gives some indication of where differences may lie 
(see Table 9). External facilitators appeared more satisfied (4.04, SD 0.96) with the relevance of 
CSRDnet to their local and state situations than did local school district central office staff (3.27, SD 
1.16), “other” respondents (3.33, SD 1.15), and school staff (3.35, SD 1.04). External facilitators 
(4.08, SD 0.84) and respondents identifying their roles as “other” (4.75, SD 0.50) rated their 
satisfaction with the adequacy of materials provided about or through CSRDnet more highly than did 
school staff (3.32, SD 1.20) and local school district central office staff (3.33, SD 0.90). 

Three statistically significant differences were found among respondents working at different 
school levels. Tukey post hoc tests revealed that respondents who worked at multiple school levels 
were more satisfied with the efficiency of CSRDnet, the completeness of information provided about 
or through CSRDnet, and the knowledge of the service provider than were respondents working at 
the high school level. 
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Role 


Relevance of service to your 
state/local situation 


Adequacy of materials 
provided 


N 


Mean 


SD 


N 


Mean 


SD 


Local school district central office staff 


15 


3.27 


1.16 


15 


3.33 


0.90 


School staff 


20 


3.35 


1.04 


19 


3.32 


1.20 


School administrator 


29 


3.90 


0.98 


29 


3.90 


1.01 


CSR model developer/staff 


1 


2.00 


N/A 


1 


2.00 


N/A 


External facilitator 


26 


4.04 


0.96 


26 


4.08 


0.84 


Other 


3 


3.33 


1.15 


4 


4.75 


0.50 



Cross-tabulations indicated that many respondents who functioned at multiple school levels 
tended to be respondents whose normal work responsibilities afford them more mobility than 
classroom teachers throughout the work day (see Table 9). Local school district staff (32.3%) and 
external facilitators (27.4%) make up more than half of those who reported working at multiple 
school levels. School administrators made up 19.4% of respondents functioning at multiple school 
levels. 



Three open-ended questions were posed on the survey. Respondents were asked to describe 
in what ways CSRD services had been useful, suggestions for improving services, and what 
assistance they needed for the coming year. For the purposes of this report, responses pertaining to 
CSRDnet were identified, coded, and tabulated. 



Eleven responses to the question about ways AEL CSRD services had been useful concerned 
CSRDnet. Interestingly, seven of the 11 replies were from external facilitators. Seven replies 
indicated that CSRDnet provided helpful information. One such participant wrote that the forum 
contained a “wealth of resources and contacts,” while another reported that it “has provided internet 
addresses of some excellent sites.” Two of the seven replies also noted that the information on 
CSRDnet was timely. One, however, noted a difficulty: “[The moderator] has a ton of information 
on the internet, but it is hard to drive the forces of a school and stay on the internet.” 



Three responses indicated that the training received about CSRDnet had been beneficial. One 
reply simply conveyed that CSRDnet had been useful. 

Nine replies about CSRDnet were given to the item soliciting suggestions to improve AEL’s 
CSRD services. Two comments indicated that CSRDnet had not been accessible during their training 
sessions, implying that training would have been strengthened had the site been available. The 
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remaining responses were idiosyncratic (see Box 1). These comments indicate issues of internet 
access, time, the voluntary nature of medium usage, site of training, format, technical difficulties, and 
limited information. 



Box 1 

Idiosyncratic Suggestions for Improving CSRDnet 



“Two of our schools are not online so CSRDnet is not working for them.” 

“CSRDnet is a wonderful concept, but so far I find it time consuming and have quit visiting the site . . . The training 
probably needs to be longer for novices like myself.” 

“I should have taken advantage of the CSRDnet more. The few times I used the web I enjoyed the experience and learned 
a great deal.” 

“Onsite delivery of services and training . . . [on] CSRDnet.” 

“The CSRDnet could be made more user friendly.” 

“CSRDnet was hard to access in the beginning. Passwords wouldn’t work. Used principal’s name. It was fixed, but there 
wasn’t much on our model.” 

“CSRDnet would have been beneficial.” 



Five responses to the query about what assistance respondents needed for the coming year 
pertained to CSRDnet. Two replies concerned training: One suggested simply offering more training 
on CSRDnet, while the second requested onsite training about the forum. One respondent wanted 
“even more relevant, research based articles on CSRDnet.” Another suggested that AEL “continue 
to seek ways to enhance the use of the CSRDnet.” Continuing the forum and allowing “non-CSRD 
schools to access the web site to help them choose effective reform models” was yet another 
suggestion. 

Listserv Activity 

Activity on the three CSRD listservs was monitored beginning January 1, 2000, and stored 
in an e-mail utility. Messages from the moderator and participants were counted. Content of postings 
from participants was coded and tabulated. 

Thirty-nine messages were posted to the Kentucky listserv, only 1 3 (33%) of which were from 
participants. Five messages (13%) debated the use of scripted lessons. Three e-mails (8%) pertained 
to a particular reform model, while two (5%) addressed methods to keep “fne fire [of reform] 
burning.” Three messages (8%) expressed thanks to the moderator for information. 

Thirty-six messages were contributed to the Virginia listserv. Eleven (31%) of these were 
from participants, five (14%) of which dealt with the use of scripted lessons. Two e-mails (6%) 
expressed thanks to the moderator, and two (6%) announced new e-mail addresses. One message 
(3%) requested technical assistance, and another (3%) discussed a reform model. 
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The West Virginia listserv had the fewest postings. Twenty-nine messages were contributed, 
only two (7%) by participants. One (50%) concerned the use of scripted lessons, the other (50%) a 
reform model. 

The moderator offered ongoing discussion prompts and responded to participant contributions 
in a timely manner in all three listservs. The moderator also provided informative links and quotes 
at least once weekly. 



CONCLUSIONS AND RECOMMENDATIONS 
Conclusions 

CSRDnet appears to be a forum which is currently underutilized. Participation of those with 
access to the medium has been minimal. Few files are accessed and fewer entries posted. On the other 
hand, AEL staff use CSRDnet more frequently. 

Respondents to the annual survey are satisfied with the format, timeliness, and efficiency of 
CSRDnet, and the knowledge and preparedness of AEL staff with regard to the forum. They are less 
satisfied with the usefulness of CSRDnet to their work, the degree to which it meets their needs, and 
its relevance to their particular situations. 

External facilitators may be more satisfied with the relevance of CRSDnet and the adequacy 
of materials they have received through or about the medium than other role groups. This may not 
be surprising, given the facilitators’ in-depth involvement with CSRD undertakings. 

Respondents working at multiple school levels are more satisfied with the efficiency of 
CSRDnet, the completeness of information provided, and the knowledge of AEL CSRDnet providers 
than are those working at the high school level. One reason for this disparity may be that respondents 
working at multiple levels have more job flexibility and physical mobility during their work day to 
find time to access CSRDnet. Another explanation may be that the organizational structure of high 
schools does not allow staff the time or latitude to explore the forum. 

Although there is little information to be gained from responses to open-ended questions on 
the annual survey with regard to CSRDnet, a few tentative conclusions may be drawn. For instance, 
external facilitators may have opportunities to use the forum more than others, and therefore may find 
it more useful as well. Also, there may be many reasons CSRDnet is underutilized; respondents’ 
comments, though few and idiosyncratic, suggest several of these, including lack of internet access, 
lack of time, the voluntary nature of CSRDnet usage, limited training opportunities, the site of 
training, format, and technical difficulties. 

Most activity on the three CSRD listservs is generated by the moderator. Of the limited 
discussion among participants via this medium, about half dealt with scripted lessons. Further, the 
Kentucky and Virginia listservs are slightly more active than the West Virginia listserv. 
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Recommendations 

CSRD staff may want to explore further the reasons CSRDnet and the listservs are 
underutilized, as well as suggestions that participants and nonparticipants have to improve their use. 
This might take the form of questionnaires, interviews, or online prompts. In addition, staff might 
consider other means by which to connect CSRD sites to each other to share information and 
resources. 

Similarly, staff should consider strategies that focus on enhancing the participation of those 
working at the high school level on CSRDnet. One such strategy might be to provide reform model 
or state information available on the forum in hard copy form to participants, citing CSRDnet as the 
source should they decide to access it for further details. 

Staff should continue to offer training on the use of CSRDnet and consider offering training 
to schools. Providing training onsite at CSRD schools may be a strategy for AEL staff to explore with 
state departments of education. External facilitators could perhaps assist with such a training effort. 

CSRD staff should continue to support the use of CSRDnet by external facilitators. One 
strategy might be to continue offering sessions on the medium at the annual External Facilitators 
Academy or at state gatherings. 

If schools selected to receive CSRD funds are indeed the lowest performing, they may not find 
accessing CSRDnet to be a priority at this time, despite its abundance of information. CSRD staff 
might consider techniques such as offering incentives for participation and hosting virtual meetings 
in chat areas to promote the usefulness of CSRDnet as an information source and a medium for 
connecting people with similar concerns. Likewise, incentives might be offered to promote listserv 
usage, particularly in West Virginia. 



O 
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1999 Survey of Client Satisfaction with AEL’s CSRD Services 
Local School District Version 

Our records indicate that you received a service or services from AEL to assist in your efforts to implement the Comprehensive 
School Reform Demonstration (CSRD) program. We are interested in continuously improving our services. Please help us 
do this by answering the following questions. Your responses will remain confidential and anonymous, so feel free to answer 
candidly.' ; ■ ../ . . ‘ .-‘X."' ' 

1 . Please check the option that best describes your role. 

Local school district central office staff 

School staff 

School administrator 

Other (please specify) 

2. In which state do you work? Please check one. 

Kentucky West Virginia Other (please specify) 

Tennessee Virginia 

3. At what school level do you work? Check all that apply. 

Elementary Jr. High Other (please specify) . 

Middle High Not applicable/available 

4. Please check the option that best represents the number of years of experience in education you have had. 

Less than 1 year 4-10 years More than 20 years 

1-3 years 1 1-20 years 

5. Check the option that best represents the number of years you have worked in your current location. 

Less than 1 year 4-10 years More than 20 years 

1-3 years 1 1-20 years Not applicable/available 

6. Please briefly describe your role in CSRD 



Comprehensive school reform model 

developer/staff 

External facilitator 



7. Listed below are services offered by AEL during 1999. Please check the services you received. Then, using the scale 
provided, rate your level of satisfaction with characteristics listed below of the AEL CSRD service(s) you indicated 
having received. 



a. 



Workshop or consultation to assist with your school’s or district’s preparation to apply for CSRD funds 



1 

1 


2 


3 


4 




5 N/A 


Very 

Dissatisfied 


Somewhat 

Dissatisfied 


Neutral 


Somewhat 

Satisfied 


Very 

Satisfied 


Not 

Applicable 
Circle One 




a. Format of service delivery 

b. Efficiency of service delivery 

c. Timeliness of service delivery 

d. Relevance of service to your state/local situation 

e. Usefulness of service to your work 

f. Adequacy of materials provided 

g. Extent to which service met your needs 

h. Completeness of information provided 

i. Knowledge of service provider 

j. Preparedness of service provider 

k. Applicability of information received 

l. Overall satisfaction with this AEL CSRD service 

BEST COPY AVAILABLE 
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12 3 4 
12 3 4 
1 2 ,3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 



b. 

model 



Workshop or consultation to assist with your school’s or district’s implementation of CSRD-funded school reform 



1 2 3 

Very Somewhat Neutral 

Dissatisfied Dissatisfied 

a. Format of service delivery 


4 

Somewhat 

Satisfied 


5 

Very 

Satisfied 


N/A 

Not 

Applicable 
Circle One 
1 2 3 


4 


5 


N/A 


b. Efficiency of service delivery 






1 2 


3 


4 


5 


N/A 


c. Timeliness of service delivery 






1 2 


3 


4 


5 


N/A 


d. Relevance of service to your state/local situation 






1 2 


3 


4 


5 


N/A 


e. Usefulness of service to your work 






1 2 


3 


4 


5 


N/A 


f. Adequacy of materials provided 






1 2 


3 


4 


5 


N/A 


g. Extent to which service met your needs 






1 2 


3 


4 


5 


N/A 


h. Completeness of information provided 






1 2 


3 


4 


5 


N/A 


i. Knowledge of service provider 






1 2 


3 


4 


5 


N/A 


j. Preparedness of service provider 






1 2 


3 


4 


5 


N/A 


k. Applicability of information received 






1 2 


3 


4 


5 


N/A 


1. Overall satisfaction with this AEL CSRD service 






1 2 


3 


4 


5 


N/A 



c. Assistance with evaluation of implementation of your CSRD-funded school reform program 



1 2 3 4 5 N/A 

Very Somewhat Neutral Somewhat Very Not 

Dissatisfied Dissatisfied Satisfied Satisfied Applicable 

Circle One 

a. Format of service delivery 1 2 3 

b. Efficiency of service delivery 1 2 3 

c. Timeliness of service delivery 1 2 3 

d. Relevance of service to your state/local situation 1 2 3 

e. Usefulness of service to your work 1 2 3 

f. Adequacy of materials provided 1 2 3 

g. Extent to which service met your needs 1 2 3 

h. Completeness of information provided 1 2 3 

i. Knowledge of service provider 1 2 3 

j. Preparedness of service provider 1 2 3 

k. Applicability of information received 1 2 3 

l. Overall satisfaction with this AEL CSRD service 1 2 3 



I 



4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 

4 5 N/A 



d. CSRDnet 

1 2 3 4 5 N/A 

Very Somewhat Neutral Somewhat Very Not 

Dissatisfied Dissatisfied Satisfied Satisfied Applicable 

Circle One 

a. Format of service delivery 1 2 3 4 5 

b. Efficiency of service delivery 1 2 3 4 5 

c. Timeliness of service delivery • • 1 2 3 4 5 

d. Relevance of service to your state/local situation 1 2 3 4 5 

e. Usefulness of service to your work 1 2 3 4 5 

f. Adequacy of materials provided 1 2 3 4 5 

g. Extent to which service met your needs 1 2 3 4 5 

h. Completeness of information provided 1 2 3 4 5 

i. Knowledge of service provider 1 2 3 4 5 

j. Preparedness of service provider 1 2 3 4 5 

k. Applicability of information received 1 2 3 4 5 

l. Overall satisfaction with this AEL CSRD service 1 2 3 4 5 



N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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e. Other (please specify) 

1 2 3 4 5 N/A 

Very Somewhat Neutral Somewhat Very Not 

Dissatisfied Dissatisfied Satisfied Satisfied Applicable 

Circle One 

a. Format of service delivery 1 2 3 4 5 N/A 

b. Efficiency of service delivery 1 2 3 4 5 N/A 

c. Timeliness of service delivery 1 2 3 4 5 N/A 

d. Relevance of service to your state/local situation 1 2 3 4 5 N/A 

e. Usefulness of service to your work 1 2 3 4 5 N/A 

f. Adequacy of materials provided 1 2 3 4 5 N/A 

g. Extent to which service met your needs 1 2 3 4 5 N/A 

h. Completeness of information provided 1 2 3 4 5 N/A 

i. Knowledge of service provider 1 2 3 4 5 N/A 

j. Preparedness of service provider 1 2 3 4 5 N/A 

k. Applicability of information received 1 2 3 4 5 N/A 

l. Overall satisfaction with this AEL CSRD service 1 2 3 4 5 N/A 



8. Please list 2 or 3 specific ways the CSRD services you have received from AEL have been useful to you. 



9. Please describe how the CSRD assistance you received could have been improved. 



10. Describe the kinds of CSRD assistance you would like to receive this year. 



Thanks for taking the time to complete this survey. Your cooperation is appreciated! Please return to Caitlin Howley- 
Rowe, AEL, Inc. 1 , Pi O. Box 1348; Charleston, WV 25325.4^ J - V ; /. 
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APPENDIX B: 



Completed Evaluation Standards Checklist 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



U1 

U2 

U3 

U4 

U5 

U6 

U7 

FI 

F2 

F3 

PI 

P2 

P3 

P4 

P5 

P6 

P7 

P8 

A1 

A2 

A3 

A4 

A5 

A6 

A7 

A8 

A9 

A10 

All 

A12 



Lptor 

Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


Evaluator Credibility 


X 








Information Scope and Selection 




X 






Values Identification 


X 








Report Clarity 


X 








Report Timeliness and Dissemination 


X 








Evaluation Impact 


X 








Practical Procedures 


X 








Political Viability 


X 








Cost Effectiveness 


X 








Service Orientation 


X 








Formal Agreements 


X 








Rights of Human Subjects 


X 








Human Interactions 


X 








Complete and Fair Assessment 


X 








Disclosure of Findings 


X 








Conflict of Interest 


X 








Fiscal Responsibility 


X 








Program Documentation 


X 








Context Analysis 


X 








Described Purposes and Procedures 


X 








Defensible Information Sources 


X 








Valid Information 


X 








Reliable Information 


X 








Systematic Information 


X 








Analysis of Quantitative Information 


X 








Analysis of Qualitative Information 


X 








Justified Conclusions 


X 








Impartial Reporting 


X 








Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

evaluation report 

other: 



Name Caitlin Howley-Rowe 






Date 



5/19/00 



(signature) 

Position or Title Research Associate 



Agency . 
Address . 



AEL , Inc . 



P.0. Box 1348 Charleston, WV 25325 



Relation to Document Author 



(e.g., author of document, evaluation team leader, external auditor, internal auditor) 



ERIC 
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